Building less-flawed metrics: Understanding and creating better measurement and incentive systems.

Metrics are useful for measuring systems and motivating behaviors in academia as well as in public policy, medicine, business, and other systems. Unfortunately, naive application of metrics to a system can distort the system and even undermine the original goal. There are two interrelated problems to overcome in building better metrics in academia and elsewhere. The first, specifying evaluable metrics that correspond to the goals, is well recognized but still often ignored. The second, minimizing perverse effects that undermine the metric or that enable people to game the rewards, is less recognized but is critical. This perspective discusses designing metrics, beginning with design considerations and processes; the presentation of specific strategies for mitigating perverse impacts, including secrecy, randomization, diversification, and post hoc specification; and continuing with important desiderata and tradeoffs involved with examples of how they can complement each other or differ. Finally, this perspective presents a comprehensive process integrating these ideas.

Managing the Transition to Widespread Metagenomic Monitoring: Policy Considerations for Future Biosurveillance.

The technological possibilities and future public health importance of metagenomic sequencing have received extensive attention, but there has been little discussion about the policy and regulatory issues that need to be addressed if metagenomic sequencing is adopted as a key technology for biosurveillance. In this article, we introduce metagenomic monitoring as a possible path to eventually replacing current infectious disease monitoring models. Many key enablers are technological, whereas others are not. We therefore highlight key policy challenges and implementation questions that need to be addressed for "widespread metagenomic monitoring" to be possible. Policymakers must address pitfalls like fragmentation of the technological base, private capture of benefits, privacy concerns, the usefulness of the system during nonpandemic times, and how the future systems will enable better response. If these challenges are addressed, the technological and public health promise of metagenomic sequencing can be realized.

A Systematic Review of Human Challenge Trials, Designs, and Safety.

BACKGROUND: Few studies have assessed participant safety in human challenge trials (HCTs). Key questions regarding HCTs include how risky such trials have been, how often adverse events (AEs) and serious adverse events (SAEs) occur, and whether risk mitigation measures have been effective. METHODS: A systematic search of PubMed and PubMed Central for articles reporting on results of HCTs published between 1980 and 2021 was performed and completed by 7 October 2021. RESULTS: Of 2838 articles screened, 276 were reviewed in full. A total of 15 046 challenged participants were described in 308 studies that met inclusion criteria; 286 (92.9%) of these studies reported mitigation measures used to minimize risk to the challenge population. Among 187 studies that reported on SAEs, 0.2% of participants experienced at least 1 challenge-related SAE. Among 94 studies that graded AEs by severity, challenge-related AEs graded "severe" were reported by between 5.6% and 15.8% of participants. AE data were provided as a range to account for unclear reporting. Eighty percent of studies published after 2010 were registered in a trials database. CONCLUSIONS: HCTs are increasingly common and used for an expanding list of diseases. Although AEs occur, severe AEs and SAEs are rare. Reporting has improved over time, though not all papers provide a comprehensive report of relevant health impacts. We found very few severe symptoms or SAEs in studies that reported them, but many HCTs did not report relevant safety data. This study was preregistered on PROSPERO as CRD42021247218.

Molecular and Serologic Diagnostic Technologies for SARS-CoV-2.

The COVID-19 pandemic has presented many challenges that have spurred biotechnological research to address specific problems. Diagnostics is one area where biotechnology has been critical. Diagnostic tests play a vital role in managing a viral threat by facilitating the detection of infected and/or recovered individuals. From the perspective of what information is provided, these tests fall into two major categories, molecular and serological. Molecular diagnostic techniques assay whether a virus is present in a biological sample, thus making it possible to identify individuals who are currently infected. Additionally, when the immune system is exposed to a virus, it responds by producing antibodies specific to the virus. Serological tests make it possible to identify individuals who have mounted an immune response to a virus of interest and therefore facilitate the identification of individuals who have previously encountered the virus. These two categories of tests provide different perspectives valuable to understanding the spread of SARS-CoV-2. Within these categories, different biotechnological approaches offer specific advantages and disadvantages. Here we review the categories of tests developed for the detection of the SARS-CoV-2 virus or antibodies against SARS-CoV-2 and discuss the role of diagnostics in the COVID-19 pandemic.

The Structure of Tweets About Vaccine Safety Between Health Organizations, Experts, and the Public: Analyzing Risk Communication Conversations.

OBJECTIVE: This article considers how health education organizations in the World Health Organization's 9 Vaccine Safety Network (VSN) use Twitter to communicate about vaccines with the public, and whether they answer questions and engage in conversations. Almost no research in public health, to our knowledge, has explored conversational structure on social media among posts sent by different accounts. METHODS: Starting with 1,017,176 tweets by relevant users, we constructed 2 corpuses of multi-tweet conversations. The first was 1,814 conversations that included VSN members directly, whereas the second was 2,283 conversations mentioning vaccines or vaccine denialism. The tweets and user metadata were then analyzed using an adaptation of rhetorical structure theory. RESULTS: In the studied data, VSN members tweeted 12,677 times within conversations, compared to their 37,587 lone tweets. Their conversations were shorter than those in the comparison corpus (P < 0.0001), and they were involved in fewer multilogues (P < 0.0001). CONCLUSION: While there is diversity among organizations, most were tied to the pre-social media broadcast model. In the future, they should try to converse more, rather than tweet more, and embrace best practices in risk communication.

Results of a 2020 Survey on Reporting Requirements and Practices for Biocontainment Laboratory Accidents.

Biosafety laboratory accidents are a normal part of laboratory science, but the frequency of such accidents is unclear due to current reporting standards and processes. To better understand accident reporting, a survey was created, with input from ABSA International, which included a series of questions about standards, requirements, and likely motivations for reporting or nonreporting. A total of 60 biosafety officers completed the survey. Respondents reported working with more than 5,000 people in laboratories, including more than 40 biosafety level 3 or animal biosafety level 3 laboratories, which work with higher-risk pathogens. Most of the respondents were located in the United States, Canada, or New Zealand, or did not identify their location. Notable results included that 97% of surveyed biosafety officers oversee laboratories that require reporting exposure to at least some pathogens. However, 63% relayed that the reports are not usually sent outside of the institution where they occurred. A slight majority (55%) stated that paper reports were used, with the rest reporting they used a variety of computer systems. Even in laboratories that used paper-based reporting systems, 67% relayed that these reports were used alongside, or entered into, a digital system. While 82% of these biosafety officers agreed that workers understood the importance of reporting for their own safety, 82% also agreed that a variety of disincentives prevent laboratory workers from reporting incidents, including concerns about job loss and loss of funding.

Exploring Risks of Human Challenge Trials For COVID-19.

Human challenge trials (HCTs) are a potential method to accelerate development of vaccines and therapeutics. However, HCTs for COVID-19 pose ethical and practical challenges, in part due to the unclear and developing risks. In this article , we introduce an interactive model for exploring some risks of a severe acute respiratory syndrome coronavirus-2 (SARS-COV-2) dosing study, a prerequisite for any COVID-19 challenge trials. The risk estimates we use are based on a Bayesian evidence synthesis model which can incorporate new data on infection fatality risks (IFRs) to patients, and infer rates of hospitalization. The model estimates individual risk, which we then extrapolate to overall mortality and hospitalization risk in a dosing study. We provide a web tool to explore risk under different study designs. Based on the Bayesian model, IFR for someone between 20 and 30 years of age is 15.1 in 100,000, with a 95% uncertainty interval from 11.8 to 19.2, while risk of hospitalization is 130 per 100,000 (100-160). However, risk will be reduced in an HCT via screening for comorbidities, selecting lower-risk population, and providing treatment. Accounting for this with stronger assumptions, we project the fatality risk to be as low as 2.5 per 100,000 (1.6-3.9) and the hospitalization risk to be 22.0 per 100,000 (14.0-33.7). We therefore find a 50-person dosing trial has a 99.74% (99.8-99.9%) chance of no fatalities, and a 98.9% (98.3-99.3%) probability of no cases requiring hospitalization.

Testing, tracing and isolation in compartmental models.

Existing compartmental mathematical modelling methods for epidemics, such as SEIR models, cannot accurately represent effects of contact tracing. This makes them inappropriate for evaluating testing and contact tracing strategies to contain an outbreak. An alternative used in practice is the application of agent- or individual-based models (ABM). However ABMs are complex, less well-understood and much more computationally expensive. This paper presents a new method for accurately including the effects of Testing, contact-Tracing and Isolation (TTI) strategies in standard compartmental models. We derive our method using a careful probabilistic argument to show how contact tracing at the individual level is reflected in aggregate on the population level. We show that the resultant SEIR-TTI model accurately approximates the behaviour of a mechanistic agent-based model at far less computational cost. The computational efficiency is such that it can be easily and cheaply used for exploratory modelling to quantify the required levels of testing and tracing, alone and with other interventions, to assist adaptive planning for managing disease outbreaks.

High-risk human-caused pathogen exposure events from 1975-2016.

Biological agents and infectious pathogens have the potential to cause very significant harm, as the natural occurrence of disease and pandemics makes clear. As a way to better understand the risk of Global Catastrophic Biological Risks due to human activities, rather than natural sources, this paper reports on a dataset of 71 incidents involving either accidental or purposeful exposure to, or infection by, a highly infectious pathogenic agent. There has been significant effort put into both reducing the risk of purposeful spread of biological weapons, and biosafety intended to prevent the exposure to, or release of, dangerous pathogens in the course of research. Despite these efforts, there are incidents of various types that could potentially be controlled or eliminated by different lab and/or bioweapon research choices and safety procedures. The dataset of events presented here was compiled during a project conducted in 2019 to better understand biological risks from anthropic sources. The events which are listed are unrelated to clinical treatment of naturally occurring outbreaks, and are instead entirely the result of human decisions and mistakes. While the events cover a wide range of cases, the criteria used covers a variety of events previously scattered across academic, policy, and other unpublished or not generally available sources.

Evaluating Use Cases for Human Challenge Trials in Accelerating SARS-CoV-2 Vaccine Development.

Human challenge trials (HCTs) have been proposed as a means to accelerate SARS-CoV-2 vaccine development. We identify and discuss 3 potential use cases of HCTs in the current pandemic: evaluating efficacy, converging on correlates of protection, and improving understanding of pathogenesis and the human immune response. We outline the limitations of HCTs and find that HCTs are likely to be most useful for vaccine candidates currently in preclinical stages of development. We conclude that, while currently limited in their application, there are scenarios in which HCTs would be extremely beneficial. Therefore, the option of conducting HCTs to accelerate SARS-CoV-2 vaccine development should be preserved. As HCTs require many months of preparation, we recommend an immediate effort to (1) establish guidelines for HCTs for COVID-19; (2) take the first steps toward HCTs, including preparing challenge virus and making preliminary logistical arrangements; and (3) commit to periodically re-evaluating the utility of HCTs.

Option-based guarantees to accelerate urgent, high-risk vaccines: a new market-shaping approach.

Accelerating the availability of COVID-19 vaccines is critical to preventing further waves and mitigating the impact on society. However, preparations for large-scale manufacturing, such as building production facilities, are typically delayed until a vaccine is proven safe and effective. This makes sense from a commercial perspective, but incurs great costs in terms of lives lost and damage to the economy. Several policy options are available to reduce this delay, all of which involve incentives or subsidies to invest in production facilities. We review existing approaches, then propose a novel alternative using "option-based guarantees" in which the government commits to paying a proportion of the manufacturer's preparation costs should the product turn out not to be viable. Counterintuitively, this "payment for failure" is appropriate because in the case of success, a company makes a profit from the product itself, and does not need additional money from the government. While other approaches have critical roles, we argue that option-based guarantees are the most promising approach to ensuring a rapid vaccine for COVID-19. Compared to the alternative approaches, they reduce both costs to the government and risk to the companies, while maintaining an incentive to produce a high-quality product quickly and at scale.

An open challenge to advance probabilistic forecasting for dengue epidemics.

A wide range of research has promised new tools for forecasting infectious disease dynamics, but little of that research is currently being applied in practice, because tools do not address key public health needs, do not produce probabilistic forecasts, have not been evaluated on external data, or do not provide sufficient forecast skill to be useful. We developed an open collaborative forecasting challenge to assess probabilistic forecasts for seasonal epidemics of dengue, a major global public health problem. Sixteen teams used a variety of methods and data to generate forecasts for 3 epidemiological targets (peak incidence, the week of the peak, and total incidence) over 8 dengue seasons in Iquitos, Peru and San Juan, Puerto Rico. Forecast skill was highly variable across teams and targets. While numerous forecasts showed high skill for midseason situational awareness, early season skill was low, and skill was generally lowest for high incidence seasons, those for which forecasts would be most valuable. A comparison of modeling approaches revealed that average forecast skill was lower for models including biologically meaningful data and mechanisms and that both multimodel and multiteam ensemble forecasts consistently outperformed individual model forecasts. Leveraging these insights, data, and the forecasting framework will be critical to improve forecast skill and the application of forecasts in real time for epidemic preparedness and response. Moreover, key components of this project-integration with public health needs, a common forecasting framework, shared and standardized data, and open participation-can help advance infectious disease forecasting beyond dengue.

Questioning Estimates of Natural Pandemic Risk.

The central argument in this article is that the probability of very large natural pandemics is more uncertain than either previous analyses or the historical record suggest. In public health and health security analyses, global catastrophic biological risks (GCBRs) have the potential to cause "sudden, extraordinary, widespread disaster," with "tens to hundreds of millions of fatalities." Recent analyses focusing on extreme events presume that the most extreme natural events are less likely than artificial sources of GCBRs and should receive proportionately less attention. These earlier analyses relied on an informal Bayesian analysis of naturally occurring GCBRs in the historical record and conclude that the near absence of such events demonstrates that they are rare. This ignores key uncertainties about both selection biases inherent in historical data and underlying causes of the nonstationary risk. The uncertainty is addressed here by first reconsidering the assumptions in earlier Bayesian analyses, then outlining a more complete analysis accounting for several previously omitted factors. Finally, relationships are suggested between available evidence and the uncertain question at hand, allowing more rigorous future estimates.

SiDMAP: a metabolomics approach to assess the effects of drug candidates on the dynamic properties of biochemical pathways.

The postgenomic era in drug development is characterised by a need to describe and predict the functional actions of a given compound within the complex systems of the organism. Recent advances in analytical and computational techniques have given rise to a new and powerful tool for the measurement of biochemical pathways in cells, animals and humans. The stable isotope dynamic metabolic profiling (SiDMAP) assay measures the flow of molecules through complex metabolic pathways, rather than just measuring the gene or protein in isolation. Thus, the SiDMAP assay is a measurement of the phenotype in biology, disease and the treatment of disease. The SiDMAP assay differs from other static approaches in two key ways: i) SiDMAP measures the activity of pathways in fully intact systems, rather than just the component pieces of the system; and ii) SiDMAP measures molecular flux observed in the dimension of time, as apposed to measuring overall levels of metabolites in a system and then trying to predict functionality. These two features confer unparalleled sensitivity to the SiDMAP analysis and have allowed for the discovery of the activity of biochemical pathways important to a number of diseases, including cancer and the metabolic syndrome and how to best treat these diseases targeting the system of pathways. Thus, SiDMAP is a technology that can be widely used in the drug discovery and development process to better describe the biochemistry of disease states, determine the method of action of compounds, detect possible toxicity early in the drug development process, reposition compounds, develop biomarkers stratify patients and to enable Phase IV studies.